<!DOCTYPE html>
<html lang="en">
    <head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta name="format-detection" content="telephone=no" />

  <title>
    27|案例篇--为什么我的磁盘I/O延迟很高？ | 迪克猪的博客
  </title>

  
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
  <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
  <link rel="manifest" href="/manifest.json" />
  <meta name="theme-color" content="#ffffff" />

  
  <link
    rel="stylesheet"
    href="https://unpkg.com/modern-normalize@0.6.0/modern-normalize.css"
  />

  
  
  
  
  <link rel="stylesheet" href="https://zsy619.github.io/style.min.f7761d111b74dd5c07f0111decee92938c12abc42e0fd319e1a07483e248b54e.css" integrity="sha256-93YdERt03VwH8BEd7O6Sk4wSq8QuD9MZ4aB0g&#43;JItU4=" />

  
  
    
  
</head>

    <body>
        <header id="header">
  <div class="header_container">
    <h1 class="sitetitle">
      <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a>
    </h1>
    <nav class="navbar">
      <ul>
        <li><a href="https://zsy619.github.io">Home</a></li>
        
          <li>
            <a href="/post/">
              
              <span>LINUX性能优化</span>
            </a>
          </li>
        
          <li>
            <a href="/csapp/">
              
              <span>深入理解计算机系统_第三版</span>
            </a>
          </li>
        
          <li>
            <a href="/golang/">
              
              <span>golang</span>
            </a>
          </li>
        
          <li>
            <a href="/docker/">
              
              <span>容器</span>
            </a>
          </li>
        
          <li>
            <a href="/flutter/">
              
              <span>Flutter</span>
            </a>
          </li>
        
          <li>
            <a href="/know/">
              
              <span>知识点</span>
            </a>
          </li>
        
          <li>
            <a href="/categories/">
              
              <span>目录</span>
            </a>
          </li>
        
          <li>
            <a href="/about/">
              
              <span>关于</span>
            </a>
          </li>
        
        <li class="hide-sm"><a href="https://zsy619.github.io/index.xml" type="application/rss+xml">RSS</a></li>
      </ul>
    </nav>
  </div>
</header>

        
<section id="main">
  <article class="post content">
    <h2 class="title">27|案例篇--为什么我的磁盘I/O延迟很高？</h2>
    <div class="post_content">
      <ul>
<li>用 iostat ，确认是否有 I/O 性能瓶颈。再用 strace 和 lsof ，来定位应用程序以及它正在写入的日志文件路径。最后通过应用程序的接口调整日志级别，完美解决 I/O 问题。</li>
</ul>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ top 
top - 14:27:02 up 10:30,  <span style="color:#ae81ff">1</span> user,  load average: 1.82, 1.26, 0.76 
Tasks: <span style="color:#ae81ff">129</span> total,   <span style="color:#ae81ff">1</span> running,  <span style="color:#ae81ff">74</span> sleeping,   <span style="color:#ae81ff">0</span> stopped,   <span style="color:#ae81ff">0</span> zombie 
%Cpu0  :  3.5 us,  2.1 sy,  0.0 ni,  0.0 id, 94.4 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu1  :  2.4 us,  0.7 sy,  0.0 ni, 70.4 id, 26.5 wa,  0.0 hi,  0.0 si,  0.0 st 
KiB Mem :  <span style="color:#ae81ff">8169300</span> total,  <span style="color:#ae81ff">3323248</span> free,   <span style="color:#ae81ff">436748</span> used,  <span style="color:#ae81ff">4409304</span> buff/cache 
KiB Swap:        <span style="color:#ae81ff">0</span> total,        <span style="color:#ae81ff">0</span> free,        <span style="color:#ae81ff">0</span> used.  <span style="color:#ae81ff">7412556</span> avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 
<span style="color:#ae81ff">12280</span> root      <span style="color:#ae81ff">20</span>   <span style="color:#ae81ff">0</span>  <span style="color:#ae81ff">103304</span>  <span style="color:#ae81ff">28824</span>   <span style="color:#ae81ff">7276</span> S  14.0  0.4   0:08.77 python 
   <span style="color:#ae81ff">16</span> root      <span style="color:#ae81ff">20</span>   <span style="color:#ae81ff">0</span>       <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span> S   0.3  0.0   0:09.22 ksoftirqd/1 
<span style="color:#ae81ff">1549</span> root      <span style="color:#ae81ff">20</span>   <span style="color:#ae81ff">0</span>  <span style="color:#ae81ff">236712</span>  <span style="color:#ae81ff">24480</span>   <span style="color:#ae81ff">9864</span> S   0.3  0.3   3:31.38 python3 

</code></pre></div><p>运行下面的 iostat 命令，其中:</p>
<ul>
<li>-d 选项是指显示出 I/O 的性能指标；</li>
<li>-x 选项是指显示出扩展统计信息（即显示所有 I/O 指标）</li>
</ul>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ iostat -d -x <span style="color:#ae81ff">1</span>
Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util 
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00 
sda              0.00   71.00      0.00  32912.00     0.00     0.00   0.00   0.00    0.00 18118.31 241.89     0.00   463.55  13.86  98.40 

</code></pre></div><p>运行下面的 pidstat 命令，观察进程的 I/O 情况：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ pidstat -d <span style="color:#ae81ff">1</span> 
14:39:14      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command 
14:39:15        <span style="color:#ae81ff">0</span>     <span style="color:#ae81ff">12280</span>      0.00 335716.00      0.00       <span style="color:#ae81ff">0</span>  python 

</code></pre></div><p>先用 strace 确认它是不是在写文件，再用 lsof 找出文件描述符对应的文件即可。</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ strace -p <span style="color:#ae81ff">12280</span> 
strace: Process <span style="color:#ae81ff">12280</span> attached 
<span style="color:#66d9ef">select</span><span style="color:#f92672">(</span>0, NULL, NULL, NULL, <span style="color:#f92672">{</span>tv_sec<span style="color:#f92672">=</span>0, tv_usec<span style="color:#f92672">=</span>567708<span style="color:#f92672">})</span> <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span> <span style="color:#f92672">(</span>Timeout<span style="color:#f92672">)</span> 
stat<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;/usr/local/lib/python3.7/importlib/_bootstrap.py&#34;</span>, <span style="color:#f92672">{</span>st_mode<span style="color:#f92672">=</span>S_IFREG|0644, st_size<span style="color:#f92672">=</span>39278, ...<span style="color:#f92672">})</span> <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span> 
stat<span style="color:#f92672">(</span><span style="color:#e6db74">&#34;/usr/local/lib/python3.7/importlib/_bootstrap.py&#34;</span>, <span style="color:#f92672">{</span>st_mode<span style="color:#f92672">=</span>S_IFREG|0644, st_size<span style="color:#f92672">=</span>39278, ...<span style="color:#f92672">})</span> <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span> 

</code></pre></div><p>借助动态追踪工具包 bcc 中的 filetop 和 opensnoop</p>

    </div>
    <div class="info post_meta">
      <time datetime=2020-06-17T08:42:18&#43;0800 class="date">Wednesday, June 17, 2020</time>
      
        <ul class="tags">
        
          <li> <a href="https://zsy619.github.io/tags/linux%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96">linux性能优化</a> </li>
        
          <li> <a href="https://zsy619.github.io/tags/linux">linux</a> </li>
        
        </ul>
      
      
    </div>
    <div class="clearfix"></div>
  </article>
  
    <div class="other_posts">
      
      <a href="https://zsy619.github.io/post/26%E6%A1%88%E4%BE%8B%E7%AF%87%E5%A6%82%E4%BD%95%E6%89%BE%E5%87%BA%E7%8B%82%E6%89%93%E6%97%A5%E5%BF%97%E7%9A%84%E5%86%85%E9%AC%BC/" class="prev">26|案例篇--如何找出狂打日志的“内鬼”？</a>
      
      
      <a href="https://zsy619.github.io/post/28%E6%A1%88%E4%BE%8B%E7%AF%87%E4%B8%80%E4%B8%AAsql%E6%9F%A5%E8%AF%A2%E8%A6%8115%E7%A7%92%E8%BF%99%E6%98%AF%E6%80%8E%E4%B9%88%E5%9B%9E%E4%BA%8B/" class="next">28|案例篇--一个SQL查询要15秒，这是怎么回事？</a>
      
    </div>
    <aside id="comments">
</aside>

  
</section>

        <a id="back_to_top" title="Go To Top" href="#">
  <span>
    <svg viewBox="0 0 24 24">
      <path fill="none" d="M0 0h24v24H0z"></path>
      <path d="M12 2L4.5 20.29l.71.71L12 18l6.79 3 .71-.71z"></path>
    </svg>
  </span>
</a>

        <footer id="footer">
  <p>
    <span>&copy; 2021 <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a> </span>
    <span>Built with <a rel="nofollow" target="_blank" href="https://gohugo.io">Hugo</a></span>
    <span>Theme by <a rel="nofollow" target="_blank" href="https://github.com/wayjam/hugo-theme-mixedpaper">WayJam</a></span>
  </p>

  <script src="https://zsy619.github.io/js/main.min.8b182175f5874aeed0acc0979345c98d4bde22208ec4f36cc1d6e3102acb4b10.js" integrity="sha256-ixghdfWHSu7QrMCXk0XJjUveIiCOxPNswdbjECrLSxA=" crossorigin="anonymous" async></script>
</footer>

    </body>
</html>
